Road plane output with lateral slope

ABSTRACT

The present disclosure generally relates to processing visual data of a road surface that includes a vertical deviation with a lateral slope. In some embodiments, a system determines a path expected to be traversed by at least one wheel of the vehicle on a road surface. In some embodiments, a system determines, using at least two images captured by one or more cameras, a height of the road surface for at least one point along the path to be traversed by the wheel. In some embodiments, a system computes an indication of a lateral slope of the road surface at the at least one point along the path. In some embodiments, a system outputs, on a vehicle interface bus, an indication of the height of the point and an indication of the lateral slope at the at least one point along the path.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/460,183, entitled “Road Plane Output with Lateral Slope,” filed onMar. 15, 2017, which claims the benefit of U.S. Provisional ApplicationNo. 62/308,631, entitled “Road Plane Output with Lateral Slope,” filedon Mar. 15, 2016, the entire contents of which are incorporated hereinby reference for all purposes.

This application is related to U.S. application Ser. No. 14/554,500 (nowU.S. Pat. No. 9,256,791), titled “Road Vertical Contour Detection,”filed Nov. 26, 2014; to U.S. application Ser. No. 14/798,575, titled“Road Contour Vertical Detection,” filed Jul. 14, 2015; to U.S.Provisional Application No. 62/120,929, titled “Road Plane ProfileOutput in a Stabilized World Coordinate Frame,” filed Feb. 26, 2015; toU.S. Provisional Application No. 62/131,374, titled “Road Plane Outputin a Stabilized World Coordinate Frame,” filed Mar. 11, 2015; to U.S.Provisional Application No. 62/149,699, titled “Road Plane ProfileOutput in a Stabilized World Coordinate Frame,” filed Apr. 20, 2015; toU.S. Provisional Application No. 62/238,980, titled “Road Plane Outputin a Stabilized World Coordinate Frame,” filed Oct. 8, 2015; and to U.S.application Ser. No. 15/055,322, titled “Road Vertical Contour DetectionUsing a Stabilized Coordinate Frame,” filed Feb. 26, 2016, the entirecontents of which are incorporated herein by reference for all purposes.

FIELD

The present disclosure relates generally to driver assistance systems,and more specifically to detecting features of a road plane.

BACKGROUND

The use of driver assistance systems (DAS) has significantly increasedin recent years, and only promises to continue. Driver assistancesystems can be hardware and/or software components that assist with thedriving or maneuvering of a vehicle. In some instances, a DAS canachieve fully autonomous control of a vehicle (e.g., no driverintervention is required during operation) or semi-autonomous control(e.g., some driver intervention is required during operation). In someinstances, a DAS can perform in tandem with driver control (e.g., makingminor corrections or providing useful information about roadconditions). In some instances, the level of control of the DAS can bevaried (e.g., by the driver) to be fully autonomous, semi-autonomous, intandem control with the driver, or disabled. Some examples of DASfunctions include lane departure warning (LDW), automatic high-beamcontrol (AHC), traffic sign recognition (TSR) and forward collisionwarning (FCW).

Many DAS systems rely on one or more cameras to capture images of thevehicle's surroundings, for example, for determining features in a roadplane (e.g., on the image plane of an imaged road surface). Sometechniques for determining features in a road plane do not correctlydetect features under certain conditions. As a result, a vehicletraveling under the control of a DAS system may not make the properadjustments to address the actual road plane features, causing adverseeffects to the ride, comfort, and/or safety of the trip.

BRIEF SUMMARY

Accordingly, the techniques provided herein allow for improveddetermination of features in a road plane, particularly features havinga lateral slope.

In some embodiments, a method for processing visual data of a roadsurface is performed, the method comprising: accessing visual datarepresenting a road surface; determining an initial road profile of theroad surface, wherein the road profile is derived from residual motionalong a vehicle path associated with the road surface; segmenting aportion of the visual data representing the road surface into a firstsegment and a second segment, wherein the portion of the visual datathat is segmented includes visual data representing an verticaldeviation on the road surface; determining a first segment road profilefor the first segment of the portion of the visual data; determining asecond segment road profile for the second segment of the portion of thevisual data; comparing one or more of the first segment road profile,the second segment road profile, and the initial road profile; and basedat least in part on results of the comparison, outputting an indicationof a lateral slope of the vertical deviation on the road surface.

In some embodiments, a system for processing visual data of a roadsurface, the system comprises: one or more processors; memory; and oneor more programs, wherein the one or more programs are stored in thememory and configured to be executed by the one or more processors, theone or more programs including instructions for: accessing visual datarepresenting a road surface; determining an initial road profile of theroad surface, wherein the road profile is derived from residual motionalong a vehicle path associated with the road surface; segmenting aportion of the visual data representing the road surface into a firstsegment and a second segment, wherein the portion of the visual datathat is segmented includes visual data representing an verticaldeviation on the road surface; determining a first segment road profilefor the first segment of the portion of the visual data; determining asecond segment road profile for the second segment of the portion of thevisual data; comparing one or more of the first segment road profile,the second segment road profile, and the initial road profile; and basedat least in part on results of the comparison, outputting an indicationof a lateral slope of the vertical deviation on the road surface.

In some embodiments, a non-transitory computer-readable storage mediumstores one or more programs, the one or more programs comprisinginstructions, which, when executed by one or more processors of avehicle, cause the processors to: access visual data representing a roadsurface; determine an initial road profile of the road surface, whereinthe road profile is derived from residual motion along a vehicle pathassociated with the road surface; segment a portion of the visual datarepresenting the road surface into a first segment and a second segment,wherein the portion of the visual data that is segmented includes visualdata representing an vertical deviation on the road surface; determine afirst segment road profile for the first segment of the portion of thevisual data; determine a second segment road profile for the secondsegment of the portion of the visual data; compare one or more of thefirst segment road profile, the second segment road profile, and theinitial road profile; and based at least in part on results of thecomparison, output an indication of a lateral slope of the verticaldeviation on the road surface.

DESCRIPTION OF THE FIGURES

FIG. 1 depicts an exemplary grid of points used to compute a homography.

FIG. 2 depicts a camera view that includes residual motion for pointsalong predicted wheel paths and plots of metric height and distance.

FIG. 3 depicts the normalized correlation score for points along a wheelpath.

FIG. 4A depicts a camera view in which an overlaid right wheel pathpasses over the center of a speed bump.

FIG. 4B depicts a camera view in which an overlaid right wheel pathpasses over the edge of a speed bump.

FIG. 5A depicts an image strip used to compute a road profile when apath is over the center of a speed bump.

FIG. 5B depicts an image strip used to compute a road profile when apath is over the edge of a speed bump.

FIG. 6 depicts a plot of residual motion of a vehicle path computed foran image strip when a path is over the center of a speed bump.

FIG. 7 depicts a camera view in which an overlaid right wheel pathpasses over the edge of a speed bump.

FIG. 8 depicts an image strip used to compute a road profile when a pathis over edge of a speed bump.

FIG. 9 depicts a segmented image strip used to compute a road profilewhen a path is over the edge of a speed bump.

FIG. 10A depicts plots of residual motion of a vehicle path computed fora whole image strip, and for the left and right parts of the stripseparately, when the path is over the center of a speed bump.

FIG. 10B depicts plots of residual motion f a vehicle path computed fora whole image strip, and for the left and right parts of the stripseparately, when the path is over the edge of a speed bump.

FIG. 11 depicts is a flow diagram illustrating an exemplary process forprocessing visual data of a road surface.

FIG. 12 depicts is a flow diagram illustrating an exemplary process forprocessing visual data of a road surface.

FIG. 13 depicts an exemplary system for processing visual data of a roadsurface.

DETAILED DESCRIPTION

The following description is presented to enable a person of ordinaryskill in the art to make and use the various embodiments. Descriptionsof specific devices, techniques, and applications are provided only asexamples. Various modifications to the examples described herein will bereadily apparent to those of ordinary skill in the art, and the generalprinciples defined herein may be applied to other examples andapplications without departing from the spirit and scope of the variousembodiments. Thus, the various embodiments are not intended to belimited to the examples described herein and shown, but are to beaccorded the scope consistent with the claims.

1. Overview

As described briefly above, driver assistance systems can rely on one ormore imaging devices to collect visual data of a road plane around thevehicle. This visual data can be used, in turn, to determine structuralproperties of the environment of the vehicle, including features of aroad surface, objects or obstacles on the road surface, or othervertical deviations of the road surface. The term “plane” as used hereinis not limited to a purely geometric plane (i.e., flat,two-dimensional), but is used in a general sense. Thus, for example, a“road plane” (which may also be referred to as a “road surface”) can besubstantially flat but include three-dimensional components.

Certain existing methods for detecting the vertical deviation of acontour of a road using a vehicle-mounted camera are known. Certainpreviously known algorithms can be summarized as follows:

-   -   1. A first pair of two consecutive frames captured by a        vehicle-mounted camera are aligned using a homography of the        road. This gives the ego-motion (rotation R and translation T)        between the frames. For example, a grid of points (e.g., as in        FIG. 1) in the image are tracked and used to compute the        homography.    -   2. A second pair of frames is then selected, the current frame        and the nearest previous frame representing a point in time from        which the vehicle has currently moved over a minimum threshold        distance. A chaining of the first pair of frames (the        consecutive frames) is used to create an initial guess of the        homography, and a more accurate homography for the second pair        of frames is then computed, and the reference plane is        determined.    -   3. The path of the vehicle is then projected onto the image        plane shown by the lines in FIG. 2). Strips along each path are        used to compute the residual motion, which gives the profile        relative to the reference plane determined. FIG. 3 shows a        normalized correlation score for a strip 31 pixels wide along        the path of the left wheel for vertical motion ±6 pixels. A        small curvature can be seen, indicated by the arrow. This        curvature represents the small residual motion indicative of a        speed bump.    -   4. Finally, the residual motion is translated into a metric        distance and height, and is combined into a multi-frame model.        The results are shown in the upper graphs in FIG. 2.

FIGS. 1-3 illustrate the use of visual data of a speed bump on a roadsurface. FIG. 1 illustrates a grid 102 comprised of 33 points, overlaidon an image, which are tracked and used to compute a homography. In someexamples, two consecutive image frames are aligned using a homography ofthe road. In other examples, non-consecutive image frames can be used.For instance, two images frames captured several frames apart (e.g.,with one or more intervening captured frames between the two imageframes) can be used to provide a larger baseline for the homography.Aligning the image frames is also referred to as warping the second(earlier) image frame towards the first. A homography can be used todetermine structure from motion data using image frames.

In some e samples, the ego motion (rotation R and translation T) of acamera between the frames is also determined using a homography. In someembodiments, the ego motion is determined using one or more sensors onthe vehicle. In some embodiments, the ego motion is determined using oneor more image processing techniques. For example, image processing caninclude computing the ego motion from two or more image frames and aroad model. In some examples, a planar (0% grade) road surface isassumed for the road model ([0,0,1] is good for normal vehicle speed andcalibrated camera height), but other road models can be used. Examplesof other road models include, but are not limited to, a 1% incliningroad, a 2% declining road, and the like. A more complex road surface canbe modeled and used, including a road model with varying road grade, aroad model of an uneven road surface, or the like. Given the road model,an image point can be projected to a 3D point on the road model.

As used herein, and unless otherwise noted, “on” a vehicle refers toattachment, placement, positioning, mounting, or the like, within avehicle, outside of a vehicle (e.g., attached to the body), or otherwisein contact with a portion of the vehicle. For example, a componentmounted on a windshield inside of the vehicle is intended to be withinthe scope of the phrase “on”, as is a component affixed or otherwisemounted to the exterior of the vehicle.

In some examples, the points in the grid can be spaced evenly in theimage, which gives more points close to the vehicle. In some examples,the points can be spaced evenly in distance which will give points moreconcentrated higher in the image. In other examples, the points areevenly spaced vertically in the image and greater weight is given to themore distant points, for example, in a least squares computation of thehomography.

In some embodiments, the system accesses two image frames, where therewas forward translation of the vehicle over a minimum threshold, anduses chaining of the consecutive frames to determine an initialhomography. A more accurate homography is then computed and thereference plane is determined. In some embodiments, the system accessesa number of image frames other than two. For example, any number ofimage frames can be used to determine the initial homography and/or themore accurate homography (also referred to as a refined homography).

FIG. 2 illustrates exemplary visual data of a road surface, captured byan imaging device of a driver assistance system. FIG. 2 includes vehiclepath data projected into the image plane of the road surface (alsoreferred to as the road plane). Vehicle paths 202 and 204 represent therespective paths of a vehicle's left and right wheels, and are depictedas dotted lines overlaid on the image. Strips along each vehicle pathare used to compute the residual motion. The residual motion can be usedto represent the profile of the road relative to an image plane (e.g., areference plane) of the imaged road surface.

A vehicle path, such as 202 and 204, is a virtual representation of anactual path on a road surface over which a vehicle has or will travel.In some embodiments, a “virtual” vehicle path has a width of one or morepixels. In some examples, the width of a vehicle path is a constantnumber of units (e.g., pixels). In other examples, a vehicle path has anon-constant width. In some embodiments, the width of a “virtual”vehicle path relates to the width of an actual path. For example, thewidth associated with an actual path can be based on an actual vehicletire width (or other point of contact between a vehicle and a roadsurface), can be based on an average tire width, can be predeterminedbased on user input, can be a center line of a tire track (independentof a tire width), or based on any other suitable value.

FIG. 3 illustrates the normalized correlation scores for an image stripthat is 31 pixels wide along the vehicle path of the left wheel (path202) for vertical motion ±6 pixels. The normalized correlation scores,such as those illustrated in FIG. 3, can indicate a deviation or shift(e.g., in number pixels) that is needed to align each given row of asecond image frame with a first image frame. In Ha 3, the darkerportions indicate higher correlation, wherein the vertical axis (e.g.,y-axis) represents rows of pixels (each 31 pixels wide), and thehorizontal axis (x-axis) represents vertical motion in pixels. A slightcurvature can be seen in the darkened portion. Straight line 302 isoverlaid on top of the normalized correlation scores, for reference.Using line 302 as a reference, the small curvature (indicated by arrow304) can be seen. This curvature indicates residual motion, and thepresence of a vertical deviation on the imaged road surface, such as acontour or an obstacle (e.g., a speed bump).

Techniques for determining the presence of road vertical contours aredescribed in more detail in U.S. application Ser. No. 14/554,500 (nowU.S. Pat. No. 9,256,791), titled “Road Vertical Contour Detection,”filed Nov. 26, 2014; U.S. application Ser. No. 14/798,575, titled “RoadContour Vertical Detection,” filed Jul. 14, 2015; and U.S. applicationSer. No. 15/055,322, titled “Road Vertical Contour Detection Using aStabilized Coordinate Frame,” filed Feb. 26, 2016, the contents of whichare incorporated herein by reference.

In some embodiments, the residual motion along one or more vehicle pathsis translated into metric distance and metric height and combined into amulti-frame model (which may also be referred to as a multi-frameprofile). A multi-frame model refers to a model that is created byaligning multiple image frames (e.g., frames representing differentmoments in time as a vehicle traverses a portion of road) with a globalmodel to combine road profiles that correspond to an overlapping portionof the roadway. In this example, exemplary road profiles for left andright wheel vehicle paths are shown in the plots in the upper portion ofFIG. 2, plot 206 and plot 208, respectively. In some embodiments, one ormore road profiles (individually or combined into a multi-frame model)are sent over a communication bus, such as a Controller Area Network(CAN) bus.

Additional description and algorithms for determining multi-frame modelsare found in U.S. application Ser. No. 14/554,500 (now U.S. Pat. No.9,256,791), titled “Road Vertical Contour Detection,” filed Nov. 26,2014; and U.S. application Ser. No. 14/798,575, titled “Road ContourVertical Detection,” filed Jul. 14, 2015, the contents of which areincorporated herein by reference.

A stabilized world reference can be used to provide a consistent roadprofile with a very compact representation denoted as a 1D profile. Alateral slope computation, as described later in this document, can beused with either representation: the full profile or the compact 1Dprofile.

A brief description of calculating a stabilized world reference isincluded below, however, further description is found in U.S.application Ser. No. 15/055,322, titled “Road Vertical Contour DetectionUsing a Stabilized Coordinate Frame,” filed Feb. 26, 2016, the contentof which is incorporated herein by reference. In some embodiments, astabilized world coordinate frame is calculated based at least on thecombination of (a) a reference plane calculated in accordance with thecurrent frame, (b) a reference plane calculated in accordance with oneor more previous frames and (c) an assumed default reference plane basedat least in part on the fixed position of the camera relative to thevehicle. The combination of those factors can, in some embodiments,yield a stabilized world coordinate frame that is used to compare framescorresponding to different steps in time in order to calculate residualmotion and recognize vertical deviations in the surface of the roadway.

A brief description of generating a profile using a 1D samplingtechnique is included below, however, further description is found inU.S. application Ser. No. 15/055,322, titled “Road Vertical ContourDetection Using a Stabilized Coordinate Frame,” filed Feb. 26, 2016, thecontent of which is incorporated herein by reference.

In some embodiments, transmitting data for a road profile for a portionof the road in front of a vehicle can require transmission of a largenumber of values. For example, the road profile output from 5 meters to12 meters in front of a vehicle, sampled every 0.05 meters, requires 140values; the number of required values is obviously even greater for alonger profile. The large numbers of values that may be required can beproblematic for the controller area network (CAN), and may requiresignificant computational resources on the receiving end. For example,in some embodiments, 200 data points may be transmitted per wheel perframe, which may require over 1 KB of data to be transmitted per frame.In some embodiments, transmitting this amount of data for every framemay be unfeasible and/or too computationally expensive for the receivingsystem.

Furthermore, some DAS and self-driving systems require the receipt ofdata at a predefined frame rate, such as 100 Hz, such that the systemcalls for one data point corresponding to a height of the road (at apredetermined distance in front of the vehicle) every 10 milliseconds.In some embodiments, sending data to a system every 10 milliseconds maybe unfeasible because it may monopolize data channels in a vehicle, suchthat other information is unable to be sent over the same data channels.

It is thus beneficial to transmit computationally efficient amounts ofdata regarding road profiles in such a manner that (a) the total amountof data sent is computationally manageable and (b) not monopolize datachannels at all points in time.

In some embodiments, a data format for sending information about roadprofiles is provided in which the road height is output at a particulardistance in front of a wheel. In some embodiments, the distance is afixed distance (e.g., 7 meters), and in some other embodiments thedistance may be dynamically determined. For example, the distance infront of the wheel may be dynamically determined in accordance with aspeed of the vehicle, increasing the reference distance at higherspeeds. In some embodiments, the distance may be set in accordance withthe distance covered by the vehicle in a given amount of time (e.g., thedistance covered by the vehicle at its current speed in 0.5 seconds).Rather than conveying an entire profile, or profile data relating to theroad height along the entire known wheel path at each time step, merelysending data corresponding to a certain fixed distance in front of thewheel allows for less total data to be transmitted, which may conservecomputational resources and bandwidth.

In some embodiments, a data format is further provided in which, insteadof just one point of data, multiple data points are transmittedsubstantially simultaneously, to effectively multiply the frame rate.For example, if the frame rate is 10 Hz, a system may in someembodiments simulate the output of data at 100 Hz by sending 10 datapoints at once, in a burst, at a true frequency of 10 Hz. Aftertransmission, the receiving component may unpack the 10 data points andconsult them one at a time, as successively required. In someembodiments, all data points corresponding to a single frame may betransmitted in a single data transmission, while in some embodimentsthey may be transmitted in multiple transmissions which number fewerthan the number of data points per frame (e.g., 7 data points per framesent in just two CAN messages). In some embodiments, fewer than 10 datapoints may be sent per burst. In some embodiments, 16 data points may besent per burst.

From the data points in the burst, a compact 1D profile can bedetermined. In some embodiments, a system may determine a distance thatthe host vehicle is anticipated to travel in the time that correspondsto the frequency at which data will be sent. In some embodiments, thesystem may sample a road profile at n points along the projected path ofa wheel, where n is the number of data points per burst. In someembodiments, the points at which the road profile is sampled may bespaced between (e.g., may span from, or may be spaced evenly between)the selected distance in front of the wheel and that distance plus theestimated distance the vehicle will travel before the next data burst issent. In some embodiments, a road profile may be sampled at points alonga path segment that extends beyond the selected distance in front of thewheel plus the estimated distance the vehicle will travel before thenext data burst is sent; in this way, there may be some overlap betweenthe portions of the road corresponding to data bursts from differentframes, creating redundancy which in some embodiments may improveaccuracy of vertical deviation calculations.

Attention is now turned to examples that illustrate specific real-worldcases that a driver assistance system can encounter. Briefly, the firstexample presented below illustrates the simple case in which a vehiclepath (e.g., of a vehicle wheel) passes over the center of a bump on theroad surface, wherein the hump has an approximately even lateral height(e.g., the entire width of the vehicle wheel will experience the heightof the bump). The second example presented below illustrates a moredifficult case for a driver assistance system in which a vehicle pathpasses over a portion of the road plane that has a lateral slope (e.g.,such as the edge of a bump or other obstacle). In this second example, atypical vehicle navigation system may simply treat the portion with thelateral slope as a simple bump (e.g., the same as in the first example),or not may not treat it as a bump at all (e.g., an average detectedheight of the bump, over the width of the vehicle wheel, will be lowerthan in the first example, which may not cause the bump to register as asignificant obstacle to the driver assistance system).

Example 1 Vehicle Path Passing Completely Over an Obstacle

FIGS. 4-6 illustrate visual data and measurements for the first example.As described above, this first example illustrates a situation in whicha driver assistance system detects that a vehicle path (e.g., of avehicle wheel) passes over the center of an obstacle (e.g., a bump)having an approximately even lateral height on both sides of the vehiclepath.

FIG. 4A illustrates visual data received by a driver assistance system.The visual data includes a representation of road surface 402, on whichthe vehicle is traveling. FIG. 4A also includes, superimposed on thevisual data, a vehicle path 404 corresponding to the right wheel of thevehicle. Included on the vehicle path 404 is a measure of the residualmotion of the road surface along the vehicle path. FIG. 4A also depictsspeed bump 406 on road surface 402. As can be seen, vehicle path 404passes completely over bump 406, which appears to have an approximatelyeven lateral height on each side of vehicle path 404. Notably, wherevehicle path 404 passes over bump 406, the overlaid residual motion onpath 404 deflects upward. This upward deflection in the graphicalrepresentation of residual motion represents the presence of a bump(e.g., or other object, feature, obstruction, or the like) on the roadsurface. The data representing the residual motion of the road surfacecan be used to compute a road profile.

In some embodiments, a road profile is the height of the road along apath where the wheels will travel. In some examples, the height is theheight relative to some reference plane. The road profile is computedfrom the residual motion, for example, by converting residual motion inpixels to a measure of distance applicable to the road surface (e.g.,centimeters). In some examples, a road profile can be expressed inpixels.

FIG. 4A also includes plot 410, depicting the residual motions of thevehicle's left wheel (top of plot 410) and of the vehicle's right wheel(bottom of plot 410). Arrow 412 indicates the portion in the residualmotion data that represents bump 406. Note the upward deflection of theresidual motion plot at arrow 412, which occurs at about a distance of 8meters (e.g., from the vehicle).

FIG. 4B is similar to FIG. 4A, but depicts the case in which the vehiclepath passes over the edge of the speed bump. FIG. 4B is included to showan analysis made by a prior driver assistance system, presented here byway of example. Plot 430 depicts the road profile determined by theprior driver assistance system. Arrow 432 indicates the presence ofspeed bump 406. When compared, the graphically-shown residual motionsindicated by arrow 412 (FIG. 4A) and arrow 432 (FIG. 4B) appear verysimilar. That is, the profile plot shows a similar bump and gives noindication that the wheel path is half on and half off the bump. Thus, adriver assistance system presented with the situations depicted ineither FIGS. 4A and 4B may treat them both the same. However, thelateral slope of the edge of the bump that is passed over in FIG. 4B maygo undetected, preventing the driver assistance system (or other vehiclecomponents) from making appropriate adjustments or taking appropriateaction. It would be appreciated that the example illustrated by FIGS. 4Aand 4B provides just one non-limiting example of a scenario whereembodiments of the present invention, and the algorithms implemented byit, react and behave differently than prior art solutions.

FIG. 5A depicts an image strip used to compute the road profile depictedin FIG. 4A. In particular, FIG. 5A shows a strip that is used fordetecting the bump, the strip being 31 pixels wide (e.g., 15 pixels oneach side of the vehicle path, plus 1 pixel representing the center ofthe vehicle path). The speed bump 502 is clearly visible for the entirewidth of the strip, beginning near row 100. In some embodiments, thelength of a strip represents a distance in front of a vehicle. Forexample, a strip can represent the visual data from a distance of 5meters in front of the vehicle to a distance of 20 meters in front of avehicle along a vehicle path. In some embodiments, this distance can beconstant. In some embodiments, this distance can be variable. Forexample, the distance represented by the length of an image strip can bevariable and depend on one or more conditions, such as vehicle speed.Similarly, the width of a strip (e.g., in pixels) can be constant orvariable. Generally, a strip with more pixels requires morecomputational resources to process.

Note that the diagonal path of vehicle path 404 in FIG. 4A has beenstraightened (e.g., to be vertical), for more efficient processing. Inaccordance with some embodiments, a vehicle path is not straightenedbefore processing. This strip can be ‘straightened’ by shifting each rowso that the center of the path forms a straight line.

Techniques related to creation and straightening of image strips aredescribed in further detail in U.S. application Ser. No. 14/554,500 (nowU.S. Pat. No. 9,256,791), titled “Road Vertical Contour Detection,”filed Nov. 26, 2014; and U.S. application Ser. No. 14/798,575, titled“Road Contour Vertical Detection,” filed Jul. 14, 2015, the contents ofwhich are incorporated herein by reference.

FIG. 5B depicts the image strip used to compute the road profiledepicted in FIG. 4B. In particular, FIG. 5B shows a strip that is usedfor detecting the bump. The speed bump covers just more than half thewidth of the strip from column 12 to column 31, however from columns 1to 12 the strip is flat road. Using the strip depicted in FIG. 5B for aroad profile computation can lead to uncertain or unsatisfactoryresults. The results can depend on the strength of the textures on theroad and the bump. If the road texture is strong the bump might appearflat. If the bump texture dominates, the result can be detection of thebump, as seen in the case of FIG. 4B. In some cases the result can be anaverage. This uncertainty is undesirable.

For example, the vehicle's experience, and its effect over a riders inthe vehicle, when the vehicle drives such a path with the wheel half onand half off of a speed bump (or an object on the road) can depend onvarious factors, including factors related to the vehicle's weightdistribution, the vehicle's suspension, the vehicle's speed, etc. Itcould be that the effect over the vehicle can be sensitive to the exactpath of the vehicle, for example, within a few centimeters. Thus, itwould be advantageous to be able to differentiate between the case wherethe path is completely on a bump or flat road and the case where thewheel path is close to the edge of an object or obstacle on the roadsurface. Optionally, the profile can be given a low confidence if itpasses on the edge of a speed bump (or any other object). In someexamples, the suspension control might have a specific response to apartial bump, and the profile estimation system can provide anappropriate indication to the suspension control system to thereby allowthe suspension control system to activate the specific partial bumpresponse. A partial bump might also give a lateral deflection force onthe car and steering control might also benefit from knowledge that sucha force is going to occur. In such a case, the profile estimation systemcan provide an appropriate indication to the steering control. If thevehicle has some form of automatic steering, the profile estimationsystem can provide an indication of a partial bump, and the automaticsteering control can be configured to subtly (as necessary) adjust thepath of the vehicle, for example, so that the wheel path does not goover the bump at all.

FIG. 6 depicts an exemplary plot of residual motion of a vehicle pathfor the example in FIG. 4A, in which the vehicle path passes over thecenter of the bump. The plot 600 depicts the residual motion of the roadplane along vehicle path 404 (measured in increments of image rowpixels). As can be seen in FIG. 6, the residual motion of the path alongthe center of the image strip depicted in FIG. 5A includes a sharpincrease that begins around pixel number 100. The resulting peakrepresents the bump.

Example 2 Vehicle Path Passing Partially Over an Obstacle

FIGS. 7-10 illustrate visual data and measurements for the secondexample. As described above, this second example illustrates a situationin which a driver assistance system detects that a vehicle path (e.g.,of a vehicle wheel) passes partially over a bump having an even lateralheight.

FIG. 7 illustrates exemplary visual data received by (or otherwisecaptured by) a driver assistance system. The visual data includes arepresentation of road surface 702, on which the vehicle is traveling,and corresponds to road surface 402. FIG. 7 also includes, superimposedon the visual data, a vehicle path 704 corresponding to the right wheelof the vehicle. Included on the vehicle path 704 is a measure of theresidual motion of the road plane along the vehicle path. FIG. 7 alsodepicts speed bump 706 on road surface 702. Bump 706 corresponds to bump406, which appears to have an approximately even lateral height.Notably, however, in this example vehicle path 704 instead passes overthe edge of bump 706. Where vehicle path 704 passes over bump 706, theoverlaid residual motion on path 704 deflects upward.

The upward deflection shown in FIG. 7 is the same upward deflectionshown in FIG. 4B, calculated according to prior techniques. As notedabove however, calculating residual motion of road features insituations such as the one presented in FIG. 7 can result in uncertainor unsatisfactory results. Embodiments of a technique for improvedprocessing of a road plane with lateral slope are presented below.

FIG. 8 illustrates an example road strip used to determine a roadprofile. FIG. 8 corresponds to the image strip of FIG. 5B, and is usedfor detecting the bump on the road surface (e.g., road surface 702). Thespeed bump covers just more than half the width of the strip from column12 to column 31, however from columns 1 to 12 the strip is flat road.Recall that in the example described with respect to FIG. 5A, theresidual motion for image strip is determined for a vehicle path passingthrough the center of the image strip.

The size of the strip (width), and of the segments described below, canbe set or can be selected according to certain parameters including forexample, the speed of the vehicle, the reaction time and otheroperational parameters of the vehicle's suspension system, the framerate of the camera, the distance at which the system is capable ofreacting (or react optimally) to detected obstacles, etc.

The longitudinal extent (length) of the strip can also be predefined orit can be selected according to certain parameters (in case ofpredefined settings, such predefined setting can also be selectedaccording to the following parameters). For example, the lateral extentof the strip can be selected according to the speed of the vehicle, thedistance to the candidate (or object) and the frame rate. Further by wayof example, the lateral extent of the strip can be selected such thateach candidate (or object) is captured at least in two (typicallyconsecutive) frames. For example, for a vehicle traveling at a speed of100 kilometers per hour, the lateral extent of the strip can be, atminimum, 150 pixels, such that the candidate obstacle will be capturedin at least strips from two consecutive image frames. It would beappreciated that the reference to two consecutive image frames as apossible parameter that can be used in determining the lateral extent ofan image frame strip is made by way of example only, and that otherlength of image frame strips can be used, include for example a lengththat would provide for capturing the candidate in three (or four, etc.)consecutive image frames.

Rather than treating the image strip of FIG. 8 as a single image stripfor processing, the image strip can be segmented. In some embodiments,the image strip is segmented into two strips. FIG. 9 depicts twoexemplary segments, a left strip (that includes pixels 0 to 15 of theimage strip) and a right segment (that includes pixels 16-31 of theimage strip). Thus, the image of FIG. 8 has been split roughly in halflongitudinally (vertically in the image) the left segment includes (foreach row width) pixels 1-15, and the right segment includes (for eachrow width) pixels 16-31. It would be appreciated that the image strip(or patch) can be segmented to any number of segments (e.g., two, three,. . . , n segments), where each segment includes two or more pixels. Insome examples, a segmented strip can be further segmented. In yetanother example, the original image strip can be segmented into aplurality of segments that partially overlap, but that do not include adirect segmentation of other segments. For example, the image strip canbe segmented into two segments of one half, as well as into threesegments of one third each. Any other appropriate segmentation of imagesis envisioned.

In some embodiments, residual motion is computed over the whole strip asbefore, but is also computed for each partial strip (segment)separately. In some embodiments, residual motion is computed for lessthan all strips. For example, residual motion can be computed for onlythe segments (partial strips).

In some embodiments, an image strip is segmented into a plurality ofsegments. In some embodiments, an image strip is segmented along alatitudinal direction. In some embodiments, an image strip is segmentedinto a plurality of segments of unequal dimensions.

In some embodiments, an image strip is segmented a plurality of times.For example, the original image may be segmented multiple times, orsegments can be further segmented. Road profiles can then be computedfor all or some of the segmented strips.

FIGS. 10A and 10B show the profile results (in units of pixels) computedfor a whole strip (“both”) and for left and right parts of the stripseparately that resulted from segmenting the whole image strip.

FIG. 10A corresponds to the second example (FIG. 8) in which the pathpasses over the edge of the bump. In this case, the left and rightsegments give distinctly different profiles. The profile for the whole(unsegmented) strip is close to the profile of the right strip which iswholly on the bump. However, the right strip and whole strip profilesare much different (e.g., have much larger values of residual motionthan) the profile for the left strip. A difference between residualmotion of the left and right strips can indicate that the vehicle's pathpasses over a lateral slope. In this example, the vehicle path passesover the edge of the speed bump, so the lateral slope is the result ofthe speed bump transitioning into flat road (e.g., when looking from theright strip to the left strip).

For the sake of comparison, FIG. 10B corresponds to the first example(FIG. 5A), in which the path passes over the center of the bump. As canbe seen, all three profiles in FIG. 10A are similar and the profile forthe whole strip effectively appears to be an average of the left andright strips.

In some embodiments, once the three profiles are computed they arecompared. The comparison can be done in units of residual motion(pixels) or the residual motion can be converted into heightmeasurements and the heights can be compared. In some embodiments,comparing profiles includes determining whether residual motion (orheight) of each of the road profiles is within a threshold value of eachother. For example, using the residual motion the system determineswhether all three measurements, for a particular row, agree within 0.25pixels. If not, in some examples a confidence associated with the roadprofile and/or residual motion of the road profile is reduced.

In some embodiments, a representation of the lateral slope of theobstacle is determined and provided as output. In some embodiments, therepresentation of the lateral slope includes a direction of the slope.In some embodiments, a bitwise representation of the slope is computed.For example, using 2 bits (0 and 1): ‘00’ means all measurements agreewithin a threshold, ‘10’ means that: (left−right)>(threshold), ‘01’means that (right−left)>(threshold), and ‘11’ might mean that theprofile for “both” (whole strip) is significantly greater or smallerthan the profiles for both “left” and “right”. The last scenario (‘11’)indicates some sort of error since typically the residual motion of“both” is somewhere in between the “left” and “right” strips, and insome examples, a confidence value may be reduced in response.

In some embodiments, after converting residual motions to metricheights, a metric difference is computed. For example, a difference in ametric height of re left strip and a metric height of the right stripprofiles is computed. In some examples, the difference is taken betweenthe two profiles at the same row of pixels along each of the leftstrip's and right strip's profiles (e.g., at approximately the samelateral distance away from the vehicle for both segments). In otherexamples, the difference computed may be the greatest difference betweenany two heights along the left and right strip profiles. A differencecan be taken between any two suitable points along the profiles of thesegmented strips.

In some embodiments, the representation of the lateral slope includes anactual slope value. In some examples, the slope value is expressed as anangle. In the example image strips depicted in FIG. 9, the centers ofthe left and right segment paths are 15 pixels apart. Given the distancefor each row (of pixels), this pixel value can be converted to actualmetric distance between image strip segment centers. For example, at adistance of 10 meters from the vehicle, a separation of 15 pixels mightbe equivalent to 10 centimeters. Given a profile height difference of 5centimeters, and a lateral distance between the centers of the twosegment paths of 9 centimeters, will mean a lateral slope of 5/9(≈20.555556). In some examples, the lateral distance used to computeslope is derived another way (e.g., not the center of two segmentpaths). Any suitable technique is intended to be within the scope ofthis disclosure. Techniques for determining metric lengths from visualdata using a homography are well known. Any suitable method fordetermining a metric height and width can be used to determine a slopevalue for a detected obstacle. In some examples, a slope value isdetermined using pixel values.

In some embodiments, the road profile system outputs all three profiles.In some embodiments, the road profile system outputs all three heightsif there is significant difference and a controller further downstreamcan make decisions. In some embodiments, fewer than all profilescomputed are output.

Efficient Computation

Various techniques can be used to increase the efficiency of computationof road profile output in accordance with the embodiments describedabove. Preferably, computation of the plurality of profiles, such as thethree profiles in the examples above (“left” strip, “right” strip, and“both”), is performed efficiently. Further, heavy computation ispreferably minimized (e.g., performed once).

In some examples, the warp of a second image frame towards a first imageframe and any preprocessing (such as rank transform, low or high passfiltering) is done once, before the strips are split. In some examples,preprocessing is performed on one or more image frames in order toreduce noise, increase textures in the images, or for any otherappropriate reason to aid the image processing techniques to beperformed using the image frames.

In some embodiments, other computations can be shared between processes.In some examples, sum of the squared difference (SSD) is used foralignment of the first and second image frames. For example, the SSDscore for a patch of 15 rows by 31 columns can be computed by adding theSSD score for the same 15 rows summed over the left columns and the SSDscore summed over the right columns. Restated, SSD scores for partialimage strips can be calculated then combined to arrive at the score forthe whole image strip, avoiding performing the SSD operation again onthe combined image strip. Thus, there is no significant additionalcomputation up to this stage. The result is three score arrays, such asshown in FIG. 3, for almost the cost of one.

In some embodiments, a peak in the normalized correlation score of aroad profile is determined first to the nearest integer and then tosubpixel using a parabolic interpolation.

The final stage of searching each row in the score array for a subpixellocal maximum is replicated three times. In some examples, some parts ofthe computation, such as computing the confidence of each row, can beskipped for the partial image strip (e.g., left strip and right strip)profiles. In some examples, computing normalized correlation instead ofSSD is similar in terms of efficiency since the vectors ΣI₁, ΣI₁ ², ΣI₂,ΣI₂ ², and ΣI₁I₂ for the whole strip are, again, simply sums of left andright. Similarly, a sum of absolute differences (SAD) computation can beused instead of SSD.

In some examples, some of the computation of subpixel local maxima canbe used if computation cycles are at a premium. The integer localmaximum for the whole strip can be used to select points for subpixelcomputation of the left and right strips. For example, subpixelcomputation can be performed for portions of a profile likely torepresent an obstacle on a road surface (e.g., local maxima).

One can also use three single frame profiles in the multi-frame profilecomputation described above. In some examples, all three measurementscan be used to compute a multi-frame median. To save computation, onecan perform the stochastic descent using only the profile from the wholestrip and align the others using same slope and offset values.

Lateral Slope Output

The determination of a lateral slope associated with a feature on a roadsurface can be used by a driver assistance system in a variety of ways.In some embodiments, one or more road profiles are provided as output.For example, the output may be provided to one or more driver assistancesystems. In some embodiments, the output includes a direction and/orvalue of a lateral slope. In some examples, the driver assistance systemcauses the adjustment of a vehicle component setting based in part on aresult of the output. Thus, comparison of the road profile of the imagestrip with the segmented image strips is used to produce actionable datafor a driver assistance system.

In some embodiments, adjustment is made to a vehicle steering setting tomodify the vehicle path, wherein the modified vehicle path avoidspassing over the bump on the road surface. For example, the vehicle maybe controlled such that it avoids having its wheel pass over the edge ofthe speed bump.

In some embodiments, adjustment is made to a vehicle suspension setting.For example, the vehicle may adjust suspension components to prepare forthe wheel to come into contact with the laterally-sloped surface, inorder to ensure safety and comfort.

In some embodiments, the driver assistance system outputs the roadprofile information and/or lateral slope information to one or morehigher-level systems that mate appropriate decisions for vehiclecontrol.

Output Over an Automotive Communication BUS

In some embodiments, the output of a system performing the techniquesdescribed herein can be communicated to the suspension controller of thevehicle of the vehicle communication BUS, such as CAN BUS or FlexRay. Itshould be appreciated that these are merely provided as examples of avehicle communication BUS, and that any other type of appropriatecommunication BUS can be used.

In the CAN BUS example, the data needs to be packaged into a series of 8byte messages. Consider now transmitting a road profile extending from5.00 m to 14.95 m. In this example, a system outputs a profile pointevery 5 cm, resulting in 200 points. The height value between −0.31 m to0.32 m at 0.0025 m resolution can be coded into 8 bits. We can reserve 4bits for a confidence value, one 1 bit for the presence of a bump, and 3bits for lateral slope. This allows for 3 values for left/right slopeand 3 values for right/left slope, 000 can indicate negligible slope and111 can indicate an error detected by the lateral slope calculation suchas residual for the full strip being outside the range of the residualmotion for the left and right partial strips.

For example, the following table (Table 1) can be used for the threeslope bits. It should be understood that Table 1 provides merely anexample of using one or more bits to represent lateral slope values, andthat any other appropriate representation of lateral slope can be used.

TABLE 1 Code Slope Size Slope Direction 000 No Slope N/A 001 Small R > L010 Medium R > L 011 Large R > L 100 Large L > R 101 Medium L > R 110Small L > R 111 Error N/A

In the example shown in Table 1, “Large” means greater than 1/1 (e.g.,the slope value), “Medium” means between 1/1 and 1/2, “Small” meansbetween 1/5 and “No Slope” means the slope is smaller than 1/5. Thus,200 points at 2 bytes each is 400 bytes or 50 CAN messages for eachprofile.

Other Techniques for Computing Lateral Slope

Any other appropriate techniques for computing an indication of lateralslope are intended to be within the scope of this disclosure. Forexample, a computed dense structure of the whole road surface could beused. Under this example, a system can: compute dense structure of thewhole road surface (e.g., using multi camera stereo or structure frommotion); determine the vehicle path along the surface; determine theprofile as the height along the path (in camera coordinates or in theroad reference plane coordinates); and determine lateral slope as thelateral change in height of points to the left and right of the path.

In some embodiments, image shear is used to compute lateral slope. Anexample of this process is provided in greater detail below.

Using Shear to Compute Lateral Slope

The lateral slope at a point along the path can be computed in analternative way using image shear. Consider a patch 17 rows and 31pixels wide centered vertically around the point. Instead of justlooking for a vertical shift v that will best align the patches derivedfrom the current to the previous images, a system can look for twoparameters: a constant v₀ for the whole patch and a parameter v_(x) thatmultiplies the x coordinate:v(x)=v ₀ +v _(x)(x)  (1)

Using Horn and Schunk Brightness Constraint (see, e.g., B. K. P. Hornand B. G. Schunck, “Determining optical flow.” Artificial Intelligence,vol. 17, pp. 185-203, 1981.):uI _(x) +vI _(v) +I _(t)=0  (2)

and setting u=0 because we align only in the vertical direction andsetting v(x)=v₀+v_(x)x where x is the column of the 31 pixel wide stripfrom x=−15 to 15, we get:(v ₀ +v _(x) x)I _(y) +I _(t)=0  (3)

To solve for the v₀ and v_(x) we can use least squares over the wholepatch following Lukas and Kanade (see, e.g., B. D. Lucas and T. Kanade(1981), An iterative image registration technique with an application tostereo vision. Proceedings of Imaging Understanding Workshop, pp.121-130.), but replace motion (u, v) in x and y (respectively) with v₀and v_(x), motion in y and shear (respectively). We find v₀ and v_(x)that minimize the SSD score over the whole patch:

$\begin{matrix}{E = {\sum\limits_{x,y}\;\left( {{\left( {\upsilon_{0} + {\upsilon_{x}x}} \right)I_{y}} + I_{t}} \right)^{2}}} & (4)\end{matrix}$

The solution is given by:

$\begin{matrix}{\begin{pmatrix}\upsilon_{0} \\\upsilon_{x}\end{pmatrix} = {{- \begin{pmatrix}{\sum\; I_{y}^{2}} & {\sum\;{x\; I_{y}^{2}}} \\{\sum\;{x\; I_{y}^{2}}} & {\sum\;{x^{2}I_{y}^{2}}}\end{pmatrix}^{- 1}}\begin{pmatrix}{\sum\;{I_{y}I_{t}}} \\{- {\sum\;{x\; I_{y}I_{t}}}}\end{pmatrix}}} & (5)\end{matrix}$

A large absolute value of v_(x) indicates a significant slope and thesign indicates a slope down from left to right or from right to left.The interpretation of v_(x) as a slope angle depends on the row of thestrip.

FIG. 11 depicts a flow diagram illustrating an exemplary process 1100for processing visual data of a road plane, in accordance with someembodiments. In some embodiments, process 1100 is performed by one ormore computing devices and/or systems (e.g., system 1200).

At block 1110, visual data representing a road surface is accessed.

At block 1120, an initial road profile of a road surface is deter Insome embodiments, the initial the road profile is derived from residualmotion along a vehicle path associated with the road surface.

At block 1130, a portion of the visual data representing the roadsurface is segmented into a first segment and a second segment. In someembodiments, the portion of the visual data that is segmented includesvisual data representing an obstacle on the road surface.

At block 1140, a first segment road profile for the first segment of theportion of the visual data is determined.

At block 1150, a second segment road profile for the second segment ofthe portion of the visual data is determined.

At block 1160, one or more of the first segment road profile, the secondsegment road profile, and the initial road profile are compared.

At block 1170, based at least in part on results of the comparison, anindication of a lateral slope of the obstacle on the road surface isoutputted.

FIG. 12 depicts a flow diagram illustrating an exemplary process 1200for processing visual data of a road plane, in accordance with someembodiments. In some embodiments, process 1200 is performed by one ormore computing devices and/or systems (e.g., system 1200).

At block 1210, a path expected to be traversed by at least one wheel ofthe vehicle on a road surface is determined.

At block 1220, using at least two images captured by the one or morecameras, a height of the road surface for at least one point along thepath to be traversed by the wheel is determined.

At block 1230, an indication of a lateral slope of the road surface atthe at least one point along the path is computed.

At block 1240, an indication of the height of the point and anindication of the lateral slope at the at least one point along the pathis output on a vehicle interface bus.

In some embodiments, using at least two images, the height of the roadat two points on the path to be traversed by the wheel, laterallydisplaced from one another and at substantially the same distance alongthe path, is determined. In some embodiments, an indication of thelateral slope of the road surface is computed based in part on thecomparing the height of the road surface at the two points.

In some embodiments, at least one of the two points is a point along acomputed road profile. In some embodiments, the two points are along twocomputed road profiles. For example, each point of the two points can beon a separate road profile.

In some embodiments, the at least one point is two or more points. Insome embodiments, the at least one point is five or more points.

In some embodiments, the at least two images were captured from a singlecamera at different times. In some embodiments, the second image iscaptured after the vehicle has moved at least a minimum distance fromthe vehicle location where the first image was captured.

In some embodiments, the minimum distance is adjustable. For example,the minimum distance from the vehicle location where the first image wascaptured can be changed based on one or more factors (e.g., vehiclespeed). In some embodiments, the minimum distance is at least 0.3meters. It should be appreciated that other minimum distances can beused.

In some embodiments, the at least two images captured were capturedusing two cameras. For example, one image may is captured by a firstcamera and a second image is captured by a second camera. In someembodiments, the two cameras are displaced laterally if the vehicle. Insome embodiments, at least one camera of the two cameras is mounted on awindshield of the vehicle.

In some embodiments, the indication of the lateral slope indicateswhether the heights to the left and right of the point are substantiallythe same. For example, the indication of the lateral slope can indicatethat adjacent points in two directions are approximately the same heightas the point.

In some embodiments, the indication of the lateral slope indicateswhether the height to the left of the point is substantially greaterthan the height to the right of the point. For example, the indicationof the lateral slope can indicate that there is a downward slope fromthe left of the point to the right of the point.

In some embodiments, the indication of the lateral slope indicateswhether the height to the left of the point is substantially smallerthan the height to the right of the point. For example, the indicationof the lateral slope can indicate that there is a downward slope fromthe right of the point to the left of the point.

In some embodiments, the indication of the lateral slope is the angle ofthe slope. In some embodiments the angle of the slope is indicated with5 or more possible values. In some embodiments, the angle of the slopeis indicated with an integer number of values. For example, the angle ofthe slop can be indicated by combinations of bits (e.g., 000, 001, 010,or the like).

FIG. 13 depicts components of an exemplary computing system 1300configured to perform any one of the above-described processes. In someembodiments, computing system 1300 is a vehicle computer, module,component, or the like. Computing system 1300 may include, for example,a processing unit including one or more processors, a memory, a storage,and input/output devices (e.g., monitor, touch screen, keyboard, camera,stylus, drawing device, disk drive, USB, Internet connection, near-fieldwireless communication, Bluetooth, etc.). However, computing system 1300may include circuitry or other specialized hardware for carrying outsome or all aspects of the processes (e.g., process 1100 and/or process1200). In some operational settings, computing system 1300 may beconfigured as a system that includes one or more units, each of which isconfigured to carry out some aspects of the processes in software,hardware, firmware, or some combination thereof.

In computing system 1300, the main system 1302 may include aninput/output (“I/O”) section 1304, one or more central processing unit(“CPU”) 1306, and a memory section 1308. Memory section 1308 may containcomputer-executable instructions and/or data for carrying out at leastportions of process 1100. The I/O section 1304 is optionally connectedto one or more camera 1310, one or more sensor 1312, a non-volatilestorage unit 1314, or one or more external system 1320. For example, anexternal system 1320 may be another vehicle control component or system,or the like. The I/O section 1304 may also be connected to othercomponents (not illustrated) that can read/write non-transitory,computer-readable storage medium, which can contain programs and/ordata.

At least some values based on the results of the above-describedprocesses can be saved for subsequent use. Additionally, anon-transitory, computer-readable storage medium can be used to store(e.g., tangibly embody) one or more computer programs for performing anyone of the above-described processes by means of a computer. Thecomputer program may be written, for example, in a general-purposeprogramming language (e.g., Pascal, C, C++, Java, or the like) or somespecialized application-specific language.

The foregoing descriptions of specific embodiments have been presentedfor purposes of illustration and description. They are not intended tobe exhaustive or to limit the scope of the claims to the precise formsdisclosed, and it should be understood that many modifications andvariations are possible in light of the above teaching.

What is claimed is:
 1. A system mounted on a host vehicle, the systemcomprising: one or more cameras; and one or more processors, wherein theone or more processors are configured to execute instructions to:determine a path expected to be traversed by at least a first wheel ofthe host vehicle on a road surface; determine, using at least two imagescaptured by one or more cameras on the host vehicle, a height of theroad surface for at least a first point located along the path expectedto be traversed by the first wheel; compute an indication of a lateralslope of the road surface at the first point; and output, on a vehicleinterface bus of the host vehicle, an indication of the height of thefirst point and an indication of the lateral slope of the road surfaceat the first point, wherein the road surface comprises the road surfaceand any stationary object on or under the road surface.
 2. The systemaccording to claim 1, wherein the one or more processors are configuredto execute instructions to cause a system response based on the lateralslope of the road surface at the first point and based on the height ofthe road surface at the first point.
 3. The system according to claim 2,wherein system response includes one of: steering the host vehicle oradjusting a suspension of the host vehicle.
 4. The system according toclaim 1, wherein to determine the height of the road surface at thefirst point, the one or more processors are configured to executeinstructions to: derive an initial road profile from a residual motionof the road surface along the path; and compute a residual motion forthe first point.
 5. The system according to claim 1, wherein the one ormore processors are configured to execute instructions to determine,using at least two images captured by the one or more cameras on thehost vehicle, a height of each one of a plurality of points locatedalong the path expected to be traversed by the first wheel.
 6. Thesystem according to claim 5, wherein at least one of the plurality ofpoints is on a stationary object on the road surface and at least one ofthe plurality of points is not on the stationary object.
 7. The systemaccording to claim 1, wherein the one or more processors are configuredto execute instructions to determine a height of the road surface at asecond point located along the path expected to be traversed by thefirst wheel, wherein the first point and the second point are laterallydisplaced from one another.
 8. The system according to claim 7, whereinthe one or more processors are configured to execute instructions tocompute a height difference between the first point and the secondpoint.
 9. The system according to claim 1, wherein the road surfaceincludes a speed bump and a pothole.
 10. The system according to claim1, wherein the one or more processors are configured to executeinstructions to determine a direction of the lateral slope.
 11. Thesystem according to claim 1, wherein the lateral slope at the firstpoint indicates an expected height difference between two or more pointsover which the first wheel is expected to be located ahead of a currentlocation of the host vehicle.
 12. A method of processing visual data ofa road surface of an environment of a host vehicle, the methodcomprising: determining a path expected to be traversed by at least afirst wheel of the host vehicle on a road surface; determining, using atleast two images captured by one or more cameras on the host vehicle, aheight of the road surface for at least a first point located along thepath expected to be traversed by the first wheel; computing anindication of a lateral slope of the road surface at the first point;and outputting, on a vehicle interface bus of the host vehicle, anindication of the height of the first point and an indication of thelateral slope of the road surface at the first point, wherein the roadsurface comprises the road surface and any stationary object on or underthe road surface.
 13. The method according to claim 12, furthercomprising causing a system response based on the lateral slope of theroad surface at the first point and based on the height of the roadsurface at the first point.
 14. The method according to claim 13,wherein system response includes one of: steering the host vehicle oradjusting a suspension of the host vehicle.
 15. The method according toclaim 12, wherein determining the height of the road surface at thefirst point comprises: deriving an initial road profile from a residualmotion of the road surface along the path; and computing a residualmotion for the first point.
 16. The method according to claim 12,further comprising determining, using at least two images captured bythe one or more cameras on the host vehicle, a height of each one of aplurality of points located along the path expected to be traversed bythe first wheel.
 17. The method according to claim 16, wherein at leastone of the plurality of points is on a stationary object on the roadsurface and at least one of the plurality of points is not on thestationary object.
 18. The method according to claim 12, furthercomprising determining a height of the road surface at a second pointlocated along the path expected to be traversed by the first wheel,wherein the first point and the second point are laterally displacedfrom one another.
 19. The method according to claim 18, furthercomprising computing a height difference between the first point and thesecond point.
 20. The method according to claim 12, wherein the roadsurface includes a speed bump and a pothole.
 21. The method according toclaim 12, further comprising determining a direction of the lateralslope.
 22. The method according to claim 12, wherein the lateral slopeat the first point indicates an expected height difference between twoor more points over which the first wheel is expected to be locatedahead of a current location of the host vehicle.
 23. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which, when executed by one ormore processors of a host vehicle, cause the processors to: determine apath expected to be traversed by at least a first wheel of the hostvehicle on a road surface; determine, using at least two images capturedby one or more cameras on the host vehicle, a height of the road surfacefor at least a first point located along the path expected to betraversed by the first wheel; compute an indication of a lateral slopeof the road surface at the first point; and output, on a vehicleinterface bus of the host vehicle, an indication of the height of thefirst point and an indication of the lateral slope of the road surfaceat the first point, wherein the road surface comprises the road surfaceand any stationary object on or under the road surface.
 24. A method ofprocessing visual data of an environment of a host vehicle, the methodcomprising: determining a path expected to be traversed by a first wheelof the host vehicle; determining, using at least two images captured byone or more cameras on the host vehicle, a height of a first pointlocated along the path to expected be traversed by the first wheel and aheight of a second point along the path to be traversed by the firstwheel, wherein the second point is a point over which the first wheel isexpected to pass while the first wheel is over the first point; andcausing a system response based on the height at the first point and theheight at the second point.
 25. The method according to claim 24,wherein the first point and the second point are laterally displacedfrom one another.
 26. The method according to claim 24, wherein thefirst point and the second point indicate an expected height differencebetween two or more points over which the first wheel is expected to belocated ahead of a current location of the host vehicle.
 27. The methodaccording to claim 24, wherein system response includes one of: steeringthe host vehicle or adjusting a suspension of the host vehicle.
 28. Themethod according to claim 24, wherein determining the height of thefirst point comprises: deriving an initial road profile from a residualmotion of a road surface along the path; and computing a residual motionfor the first point.
 29. The method according to claim 24, wherein atleast one of the first or second points is on a stationary object on aroad surface and at least one of the first or second points is not onthe stationary object.
 30. The method according to claim 29, wherein theroad surface includes a speed bump and a pothole.
 31. A system mountedon a host vehicle, the system comprising: one or more cameras; and oneor more processors, wherein the one or more processors are configured toexecute instructions to: determine a path expected to be traversed by afirst wheel of the host vehicle; determine, using at least two imagescaptured by one or more cameras on the host vehicle, a height of a firstpoint located along the path to expected be traversed by the first wheeland a height of a second point along the path to be traversed by thefirst wheel, wherein the second point is a point over which the firstwheel is expected to pass while the first wheel is over the first point;and cause a system response based on the height at the first point andthe height at the second point.
 32. The system according to claim 31,wherein the first point and the second point are laterally displacedfrom one another.
 33. The system according to claim 31, wherein thefirst point and the second point indicate an expected height differencebetween two or more points over which the first wheel is expected to belocated ahead of a current location of the host vehicle.
 34. The systemaccording to claim 31, wherein system response includes one of: steeringthe host vehicle or adjusting a suspension of the host vehicle.
 35. Thesystem according to claim 31, wherein to determine the height of thefirst point, the one or more processors are configured to executeinstructions to: derive an initial road profile from a residual motionof a road surface along the path; and compute a residual motion for thefirst point.
 36. The system according to claim 31, wherein at least oneof the first or second points is on a stationary object on a roadsurface and at least one of the first or second points is not on thestationary object.
 37. The system according to claim 36, wherein theroad surface includes a speed bump and a pothole.
 38. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which, when executed by one ormore processors of a host vehicle, cause the processors to: determine apath expected to be traversed by a first wheel of the host vehicle;determine, using at least two images captured by one or more cameras onthe host vehicle, a height of a first point located along the path toexpected be traversed by the first wheel and a height of a second pointalong the path to be traversed by the first wheel, wherein the secondpoint is a point over which the first wheel is expected to pass whilethe first wheel is over the first point; and cause a system responsebased on the height at the first point and the height at the secondpoint.